{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.2"
    },
    "colab": {
      "name": "easytransfer-quick_start.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4srj7gBeVvLL",
        "colab_type": "text"
      },
      "source": [
        "## 本教程可以直接在PAI-DSW(https://dsw-dev.data.aliyun.com/) 运行，出于安全性考虑，先在cpu环境下运行，把数据下载完毕，然后在gpu环境下运行即可, 也可以在colab（https://colab.research.google.com/） 运行"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "uuid": "783b555e-054c-4e47-953d-e03cfa53c6da",
        "id": "AFQ9M2pjSPUo",
        "colab_type": "text"
      },
      "source": [
        "# 使用Jupyter-Notebook快速搭建CLUE分类任务(https://github.com/CLUEbenchmark/CLUE) 应用\n",
        "\n",
        "这是一篇介绍如何在PAI-DSW里用EasyTransfer平台训练CLUE分类任务的教程。只需要一份配置文件，一份ipynb文件，您就可以完成对原始数据的特征提取，网络构建，损失函数及分类评估/预测的简单调用。运行本DEMO需要如下的配置信息\n",
        "\n",
        "- python 3.6+\n",
        "- tensorflow 1.12+\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "43d586bf-1963-49d6-b0e3-2e4450d9aa97",
        "id": "D29rxK_lSPUt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 632
        },
        "outputId": "7f26f509-426f-4921-8e44-f0ab980ee596"
      },
      "source": [
        "# 安装 tensorflow-gpu 1.13.1 \n",
        "!pip install tensorflow-gpu==1.13.1\n",
        "!pip install easytransfer"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting tensorflow-gpu==1.13.1\n",
            "  Using cached https://files.pythonhosted.org/packages/7b/b1/0ad4ae02e17ddd62109cd54c291e311c4b5fd09b4d0678d3d6ce4159b0f0/tensorflow_gpu-1.13.1-cp36-cp36m-manylinux1_x86_64.whl\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.18.5)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (0.35.1)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.15.0)\n",
            "Requirement already satisfied: tensorboard<1.14.0,>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.13.1)\n",
            "Requirement already satisfied: tensorflow-estimator<1.14.0rc0,>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.13.0)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (0.3.3)\n",
            "Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (0.10.0)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (3.12.4)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.0.8)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.1.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.1.2)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (1.32.0)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.1) (0.8.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow-gpu==1.13.1) (3.2.2)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow-gpu==1.13.1) (1.0.1)\n",
            "Requirement already satisfied: mock>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow-gpu==1.13.1) (4.0.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow-gpu==1.13.1) (50.3.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==1.13.1) (2.10.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->tensorflow-gpu==1.13.1) (1.7.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<1.14.0,>=1.13.0->tensorflow-gpu==1.13.1) (3.1.0)\n",
            "Installing collected packages: tensorflow-gpu\n",
            "Successfully installed tensorflow-gpu-1.13.1\n",
            "Collecting easytransfer\n",
            "  Using cached https://files.pythonhosted.org/packages/0b/0a/87741b5373072b308518c91e913093129789cbb8e474a195df02a6c0d77f/easytransfer-0.1.1-py3-none-any.whl\n",
            "Requirement already satisfied: joblib==0.14.1 in /usr/local/lib/python3.6/dist-packages (from easytransfer) (0.14.1)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from easytransfer) (0.1.91)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from easytransfer) (0.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from easytransfer) (1.18.5)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from easytransfer) (1.15.0)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->easytransfer) (0.22.2.post1)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->easytransfer) (1.4.1)\n",
            "Installing collected packages: easytransfer\n",
            "Successfully installed easytransfer-0.1.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cellType": "markdown",
        "uuid": "a6e2be90-bba3-4ddf-b694-4ed0443eff24",
        "id": "w8BUyCsbSPUw",
        "colab_type": "text"
      },
      "source": [
        "## （一）数据准备\n",
        "将训练CLUE分类任务相关数据下载到相应的文件夹，以WSC为例"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "d0558da5-a103-42df-9148-4585a7dfca19",
        "id": "H87xHYnzSPUw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "a817fc30-f470-43d4-a252-193e547a2784"
      },
      "source": [
        "!mkdir data\n",
        "!wget  https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/clue_glue_superglue_benchmark/clue_datasets.tgz\n",
        "!tar -zxf clue_datasets.tgz\n",
        "%mv clue_datasets/* data\n",
        "%rm -rf clue_datasets*"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mkdir: cannot create directory ‘data’: File exists\n",
            "--2020-09-24 14:00:12--  https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/clue_glue_superglue_benchmark/clue_datasets.tgz\n",
            "Resolving atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)... 106.14.228.37\n",
            "Connecting to atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)|106.14.228.37|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 91088579 (87M) [application/octet-stream]\n",
            "Saving to: ‘clue_datasets.tgz’\n",
            "\n",
            "clue_datasets.tgz    14%[=>                  ]  12.76M  61.2KB/s    eta 13m 20s^C\n",
            "\n",
            "gzip: stdin: unexpected end of file\n",
            "tar: Unexpected EOF in archive\n",
            "tar: Unexpected EOF in archive\n",
            "tar: Error is not recoverable: exiting now\n",
            "mv: cannot move 'clue_datasets/CMNLI' to 'data/CMNLI': Directory not empty\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cO9cOKInbXPU",
        "colab_type": "text"
      },
      "source": [
        "## （二）指定任务的名字task name （CLUE---> AFQMC, CMNLI, CSL, IFLYTEK, TNEWS, CLUEWSC)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "db6677dd-4eb5-4fc7-9c5d-b5a119bd0a8f",
        "id": "wSa53f41SPU0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "task_name=\"CLUEWSC\"\n",
        "task_dir=\"./data/\" + task_name"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ibk9UGSLos2",
        "colab_type": "text"
      },
      "source": [
        "##（三）指定预训练模型的名字  参考（https://www.yuque.com/easytransfer/cn/oszcof?inner=pqfci）"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "66e3049e-0dd8-4b6c-a171-ab3cf44e12ca",
        "id": "lpl95RlYSPU2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pretrain_model_name_or_path=\"google-bert-base-zh\""
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "bd7b9a5d-0458-4005-9c76-0b7b7b9262ca",
        "id": "FtVuleynSPU5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_data = task_dir + \"/train.csv\"\n",
        "dev_data = task_dir + \"/dev.csv\"\n",
        "test_data = task_dir + \"/test.csv\""
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RzkK4ftqUGOS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_dir = task_name + \"_model_dir\""
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cellType": "code",
        "uuid": "f56c21fb-bba9-472f-97f9-b390e7e60a82",
        "id": "F038RtpYSPU7",
        "colab_type": "text"
      },
      "source": [
        "## （四）定义配置文件\n",
        "\n",
        "如下是我们easytransfe的配置，比如说predict_checkpoint_path是指定验证集上指标最好的checkpoint的路径。\n",
        "详细配置介绍请看easytransfer文档: https://yuque.antfin-inc.com/pai/transfer-learning/zyib3t"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "fc8373fe-c7ad-46ef-8584-41a20b7846c0",
        "id": "DwIlwB29SPU7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config_json = {\n",
        "        \"worker_hosts\": \"localhost\",\n",
        "        \"task_index\": 1,\n",
        "        \"job_name\": \"chief\",\n",
        "        \"num_gpus\": 1,\n",
        "        \"num_workers\": 1,\n",
        "        \"preprocess_config\": {\n",
        "            \"input_schema\": None,\n",
        "            \"sequence_length\": 128,\n",
        "            \"first_sequence\": None,\n",
        "            \"second_sequence\": None,\n",
        "            \"label_name\": \"label\",\n",
        "            \"label_enumerate_values\": None,\n",
        "        },\n",
        "        \"model_config\": {\n",
        "            \"pretrain_model_name_or_path\": pretrain_model_name_or_path,\n",
        "            \"num_labels\": None\n",
        "        },\n",
        "        \"train_config\": {\n",
        "            \"train_input_fp\": train_data,\n",
        "            \"train_batch_size\": 16,\n",
        "            \"model_dir\": model_dir,\n",
        "            \"num_epochs\": 2,\n",
        "            \n",
        "            \"keep_checkpoint_max\": 11,\n",
        "            \"save_steps\": None,\n",
        "            \"optimizer_config\": {\n",
        "                \"optimizer\": \"adam\",\n",
        "                \"weight_decay_ratio\": 0.01,\n",
        "                \"warmup_ratio\": 0.1,\n",
        "                \"learning_rate\": 1e-5,\n",
        "            },\n",
        "            \"distribution_config\": {\n",
        "                \"distribution_strategy\": None,\n",
        "            }\n",
        "        },\n",
        "        \"evaluate_config\": {\n",
        "            \"eval_input_fp\": dev_data,\n",
        "            \"eval_batch_size\": 8\n",
        "        }\n",
        "    }"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IkkZNLgDUUsp",
        "colab_type": "text"
      },
      "source": [
        "## 定义各个任务的特定的配置"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "ed0cb97b-7c98-4df1-8a2c-50b674a8bf4d",
        "id": "7gk7oTGGSPU9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def task_config_json(val):\n",
        "    if val == \"TNEWS\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"115,114,108,109,116,110,113,112,102,103,100,101,106,107,104\"\n",
        "        config_json['model_config']['num_labels'] = 15\n",
        "    elif val == \"AFQMC\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"0,1\"\n",
        "        config_json['model_config']['num_labels'] = 2\n",
        "    elif val == \"IFLYTEK\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \",\".join([str(idx) for idx in range(119)])\n",
        "        config_json['model_config']['num_labels'] = 119\n",
        "    elif val == \"CMNLI\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"entailment,neutral,contradiction\"\n",
        "        config_json['model_config']['num_labels'] = 3\n",
        "    elif val == \"CSL\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"0,1\"\n",
        "        config_json['model_config']['num_labels'] = 2\n",
        "    elif val == \"WSC\" or val == \"CLUEWSC\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"idx:str:1,sent1:str:1,label:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"True,False\"\n",
        "        config_json['model_config']['num_labels'] = 2\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "9ecc2755-56e6-4817-9473-2fc6f1f3c9da",
        "id": "ZCxAt6_MSPU_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "task_config_json(task_name)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "uuid": "36548664-60ee-4c16-adeb-731f9104f307",
        "id": "qluQjp-hSPVB",
        "colab_type": "text"
      },
      "source": [
        "##  （五）定义分类应用\n",
        "\n",
        "### 导入ez_transfer库文件\n",
        "- base_model: 所有应用都需要继承的父类\n",
        "- Config：用来解析配置文件的父类\n",
        "- layers：基础组件。比如Embedding，Attention等\n",
        "- model_zoo: 管理预训练模型的组件库，通过get_pretrained_model方法可调用bert模型\n",
        "- preprocessors：管理各种应用的预处理逻辑\n",
        "- CSVReader：csv格式的数据读取器\n",
        "- softmax_cross_entropy：用于分类任务的损失函数\n",
        "- classification_eval_metrics：用于分类任务的评估指标，比如Accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "3a513116-1f95-4db9-919f-4543a9645776",
        "id": "_iyFZ0NUSPVB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "67cc3958-cea4-4d86-a0a2-d799ebda7109"
      },
      "source": [
        "import sys\n",
        "\n",
        "import os\n",
        "import tensorflow as tf\n",
        "\n",
        "from easytransfer import base_model, Config, FLAGS\n",
        "from easytransfer import layers\n",
        "from easytransfer import model_zoo\n",
        "from easytransfer import preprocessors\n",
        "from easytransfer.datasets import CSVReader\n",
        "from easytransfer.evaluators import classification_eval_metrics\n",
        "from easytransfer.losses import softmax_cross_entropy"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
            "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "uuid": "09da5b10-35c0-4774-a2ba-5b8e79c717c5",
        "id": "FUCqkK_vSPVE",
        "colab_type": "text"
      },
      "source": [
        "## 构图\n",
        "完整的训练/评估/预测/链路，由四个函数构成\n",
        "- build_logits: 构图\n",
        "- build_loss：定义损失函数\n",
        "- build_eval_metrics：定义评估指标\n",
        "- build_predictions：定义预测输出"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "4a9018b0-698f-4539-ae42-551326bdc9ba",
        "id": "qObe3BYRSPVE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Application(base_model):\n",
        "    def __init__(self, **kwargs):\n",
        "        super(Application, self).__init__(**kwargs)\n",
        "        self.user_defined_config = kwargs[\"user_defined_config\"]\n",
        "\n",
        "    def build_logits(self, features, mode=None):\n",
        "        preprocessor = preprocessors.get_preprocessor(self.pretrain_model_name_or_path,\n",
        "                                                      user_defined_config=self.user_defined_config)\n",
        "\n",
        "        model = model_zoo.get_pretrained_model(self.pretrain_model_name_or_path)\n",
        "\n",
        "        dense = layers.Dense(self.num_labels,\n",
        "                             kernel_initializer=layers.get_initializer(0.02),\n",
        "                             name='dense')\n",
        "\n",
        "\n",
        "        input_ids, input_mask, segment_ids, label_ids = preprocessor(features)\n",
        "\n",
        "        outputs = model([input_ids, input_mask, segment_ids], mode=mode)\n",
        "        pooled_output = outputs[1]\n",
        "\n",
        "        if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "            pooled_output = tf.nn.dropout(pooled_output, keep_prob=0.9)\n",
        "\n",
        "        logits = dense(pooled_output)\n",
        "\n",
        "        if mode == tf.estimator.ModeKeys.PREDICT:\n",
        "            return logits\n",
        "\n",
        "        return logits, label_ids\n",
        "\n",
        "    def build_loss(self, logits, labels):\n",
        "        return softmax_cross_entropy(labels, self.num_labels, logits)\n",
        "\n",
        "    def build_eval_metrics(self, logits, labels):\n",
        "        \n",
        "        return classification_eval_metrics(logits, labels, self.num_labels)\n",
        "\n",
        "    def build_predictions(self, output):\n",
        "        logits = output\n",
        "        predictions = dict()\n",
        "        predictions[\"logits\"] = logits\n",
        "        predictions[\"predictions\"] = tf.argmax(logits, axis=-1, output_type=tf.int32)\n",
        "        return predictions"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "uuid": "e12059b0-28de-42ab-9e3b-092c64f11864",
        "id": "6aWr8fXlSPVG",
        "colab_type": "text"
      },
      "source": [
        "# （六）启动训练"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "fb6deba1-38b4-43d9-9d98-40e16f552f50",
        "id": "tmatjbf5SPVG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b98ca928-485f-403d-ee79-959b5464c6f9"
      },
      "source": [
        "config = Config(mode=\"train_and_evaluate_on_the_fly\", config_json=config_json)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:***************** modelZooBasePath /root/.eztransfer_modelzoo ***************\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "11c249ab-d309-4864-b89a-4dd8be09eb1d",
        "id": "pBC9BnNsSPVI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 666
        },
        "outputId": "32169f80-764d-4ffb-bdf8-949820798c58"
      },
      "source": [
        "app = Application(user_defined_config=config)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:total number of training examples 1244\n",
            "INFO:tensorflow:***********Running in train_and_evaluate_on_the_fly mode***********\n",
            "INFO:tensorflow:***********Disable Tao***********\n",
            "INFO:tensorflow:***********Disable AUTO_MIXED_PRECISION***********\n",
            "INFO:tensorflow:***********NCCL_MAX_NRINGS 4***********\n",
            "INFO:tensorflow:***********NCCL_MIN_NRINGS 4***********\n",
            "INFO:tensorflow:***********TF_JIT_PROFILING False***********\n",
            "INFO:tensorflow:***********PAI_ENABLE_HLO_DUMPER False***********\n",
            "INFO:tensorflow:***********Single worker, Single gpu, Don't use distribution strategy***********\n",
            "INFO:tensorflow:model_dir: ./data/CLUEWSC_model_dir\n",
            "INFO:tensorflow:num workers: 1\n",
            "INFO:tensorflow:num gpus: 1\n",
            "INFO:tensorflow:learning rate: 1e-05\n",
            "INFO:tensorflow:train batch size: 16\n",
            "INFO:tensorflow:global batch size: 16\n",
            "INFO:tensorflow:num accumulated batches: 1\n",
            "INFO:tensorflow:num model replica: 1\n",
            "INFO:tensorflow:num train examples per epoch: 1244\n",
            "INFO:tensorflow:num epochs: 2.0\n",
            "INFO:tensorflow:train steps: 156\n",
            "INFO:tensorflow:save steps: 77\n",
            "INFO:tensorflow:throttle secs: 100\n",
            "INFO:tensorflow:keep checkpoint max: 11\n",
            "INFO:tensorflow:warmup ratio: 0.1\n",
            "INFO:tensorflow:gradient clip: True\n",
            "INFO:tensorflow:log step count steps: 100\n",
            "INFO:tensorflow:Using config: {'_model_dir': './data/CLUEWSC_model_dir', '_tf_random_seed': 123123, '_save_summary_steps': 100, '_save_checkpoints_steps': 77, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1024\n",
            "inter_op_parallelism_threads: 1024\n",
            "gpu_options {\n",
            "  per_process_gpu_memory_fraction: 1.0\n",
            "  allow_growth: true\n",
            "  force_gpu_compatible: true\n",
            "}\n",
            "allow_soft_placement: true\n",
            ", '_keep_checkpoint_max': 11, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe728895fd0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
            "WARNING:tensorflow:Estimator's model_fn (<function EzTransEstimator._build_model_fn.<locals>.model_fn at 0x7fe7288a1400>) includes params argument, but params are not passed to Estimator.\n",
            "INFO:tensorflow:num eval steps: None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "e3b6f209-ac71-41bd-895e-81138ad8f8d3",
        "id": "8wi1L5ncSPVK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "5fa299b1-6169-4470-cd32-61aab0b62f3b"
      },
      "source": [
        "train_reader = CSVReader(input_glob=app.train_input_fp,\n",
        "                         is_training=True,\n",
        "                         input_schema=app.input_schema,\n",
        "                         batch_size=app.train_batch_size)\n",
        "\n",
        "eval_reader = CSVReader(input_glob=app.eval_input_fp,\n",
        "                        is_training=False,\n",
        "                        input_schema=app.input_schema,\n",
        "                        batch_size=app.eval_batch_size)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:num_parallel_batches 1\n",
            "INFO:tensorflow:shuffle_buffer_size None\n",
            "INFO:tensorflow:prefetch_buffer_size 1\n",
            "INFO:tensorflow:batch_size 16\n",
            "INFO:tensorflow:distribution_strategy None\n",
            "INFO:tensorflow:num_micro_batches 1\n",
            "INFO:tensorflow:input_schema idx:str:1,sent1:str:1,label:str:1\n",
            "INFO:tensorflow:./data/CLUEWSC/train.csv, total number of training examples 1244\n",
            "INFO:tensorflow:num_parallel_batches 1\n",
            "INFO:tensorflow:shuffle_buffer_size None\n",
            "INFO:tensorflow:prefetch_buffer_size 1\n",
            "INFO:tensorflow:batch_size 8\n",
            "INFO:tensorflow:distribution_strategy None\n",
            "INFO:tensorflow:num_micro_batches 1\n",
            "INFO:tensorflow:input_schema idx:str:1,sent1:str:1,label:str:1\n",
            "INFO:tensorflow:./data/CLUEWSC/dev.csv, total number of eval examples 304\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "1bda8a70-fb92-4b8d-9c35-97c4fe1e1df6",
        "id": "LcVXf1o9SPVM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "20238658-1f2f-419f-b923-f7eb407c8cd7"
      },
      "source": [
        "app.run_train(reader=train_reader)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "INFO:tensorflow:Random shuffle on the whole 1244 training examples\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "\n",
            "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/easytransfer/preprocessors/preprocessor.py:299: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "tf.py_func is deprecated in TF V2. Instead, use\n",
            "    tf.py_function, which takes a python function which manipulates tf eager\n",
            "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
            "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
            "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
            "    being differentiable using a gradient tape.\n",
            "    \n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "INFO:tensorflow:*******Warmup 15 steps***********\n",
            "INFO:tensorflow:*******Using adamW optimizer************\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "INFO:tensorflow:*******Num of trainable variables 102883980************\n",
            "INFO:tensorflow:*******Clip Gradients************\n",
            "INFO:tensorflow:*******Clip Norm Value 1.0*********\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-0\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1070: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file utilities to get mtimes.\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 0 into ./data/CLUEWSC_model_dir/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.73896223, step = 0\n",
            "INFO:tensorflow:progress = 0.00%, avg_loss = 0.738962\n",
            "INFO:tensorflow:Saving checkpoints for 77 into ./data/CLUEWSC_model_dir/model.ckpt.\n",
            "INFO:tensorflow:global_step/sec: 2.33076\n",
            "INFO:tensorflow:loss = 0.5622012, step = 100 (42.908 sec)\n",
            "INFO:tensorflow:progress = 64.10%, avg_loss = 0.698267\n",
            "INFO:tensorflow:Saving checkpoints for 154 into ./data/CLUEWSC_model_dir/model.ckpt.\n",
            "INFO:tensorflow:Saving checkpoints for 156 into ./data/CLUEWSC_model_dir/model.ckpt.\n",
            "INFO:tensorflow:Loss for final step: 0.6170554.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "363c3e0f-d601-42a5-aead-1c353fdafd7d",
        "id": "qhE-p7YMSPVO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 938
        },
        "outputId": "54a3d382-30c3-4e86-c4dc-ca560405d626"
      },
      "source": [
        "ckpts = set()\n",
        "with tf.gfile.GFile(os.path.join(app.config.model_dir, \"checkpoint\"), mode='r') as reader:\n",
        "    for line in reader:\n",
        "        line = line.strip()\n",
        "        line = line.replace(\"oss://\", \"\")\n",
        "        ckpts.add(int(line.split(\":\")[1].strip().replace(\"\\\"\", \"\").split(\"/\")[-1].replace(\"model.ckpt-\", \"\")))\n",
        "\n",
        "# early stopping\n",
        "best_acc = 0\n",
        "best_ckpt = None\n",
        "for ckpt in sorted(ckpts):\n",
        "    checkpoint_path = os.path.join(app.config.model_dir, \"model.ckpt-\" + str(ckpt))\n",
        "    tf.logging.info(\"checkpoint_path is {}\".format(checkpoint_path))\n",
        "    eval_results = app.run_evaluate(reader=eval_reader, checkpoint_path=checkpoint_path)\n",
        "    acc = eval_results['py_accuracy']\n",
        "    if acc > best_acc:\n",
        "        best_ckpt = ckpt\n",
        "        best_acc = acc\n",
        "tf.logging.info(\"best ckpt {}, best acc {}\".format(best_ckpt, best_acc))\n",
        "best_ckpt_path=os.path.join(app.config.model_dir, \"model.ckpt-\" + str(best_ckpt))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:checkpoint_path is ./data/CLUEWSC_model_dir/model.ckpt-0\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "INFO:tensorflow:empty data to evaluate\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-09-24T14:05:47Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-0\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-09-24-14:06:05\n",
            "INFO:tensorflow:Saving dict for global step 0: global_step = 0, loss = 0.6809694, py_accuracy = 0.5888158, py_macro_f1 = 0.50188106, py_micro_f1 = 0.5888158, py_weighted_f1 = 0.5580121\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 0: ./data/CLUEWSC_model_dir/model.ckpt-0\n",
            "INFO:tensorflow:checkpoint_path is ./data/CLUEWSC_model_dir/model.ckpt-77\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "INFO:tensorflow:empty data to evaluate\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-09-24T14:06:13Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-77\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-09-24-14:06:28\n",
            "INFO:tensorflow:Saving dict for global step 77: global_step = 77, loss = 0.66377115, py_accuracy = 0.62171054, py_macro_f1 = 0.44661653, py_micro_f1 = 0.62171054, py_weighted_f1 = 0.53057975\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 77: ./data/CLUEWSC_model_dir/model.ckpt-77\n",
            "INFO:tensorflow:checkpoint_path is ./data/CLUEWSC_model_dir/model.ckpt-154\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "INFO:tensorflow:empty data to evaluate\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-09-24T14:06:34Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-154\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-09-24-14:06:40\n",
            "INFO:tensorflow:Saving dict for global step 154: global_step = 154, loss = 0.66772, py_accuracy = 0.5888158, py_macro_f1 = 0.49505016, py_micro_f1 = 0.5888158, py_weighted_f1 = 0.5537431\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 154: ./data/CLUEWSC_model_dir/model.ckpt-154\n",
            "INFO:tensorflow:checkpoint_path is ./data/CLUEWSC_model_dir/model.ckpt-156\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "INFO:tensorflow:empty data to evaluate\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-09-24T14:06:46Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-156\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-09-24-14:06:52\n",
            "INFO:tensorflow:Saving dict for global step 156: global_step = 156, loss = 0.66782266, py_accuracy = 0.5888158, py_macro_f1 = 0.49505016, py_micro_f1 = 0.5888158, py_weighted_f1 = 0.5537431\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 156: ./data/CLUEWSC_model_dir/model.ckpt-156\n",
            "INFO:tensorflow:best ckpt 77, best acc 0.6217105388641357\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5vSGPg89U49a",
        "colab_type": "text"
      },
      "source": [
        "## (七）定义预测的配置文件"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "a20c4310-5c08-4732-844f-eef6fb112532",
        "id": "rQX9vxD9SPVQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "predict_config_json = {\n",
        "        \"worker_hosts\": \"localhost\",\n",
        "        \"task_index\": 1,\n",
        "        \"job_name\": \"chief\",\n",
        "        \"num_gpus\": 1,\n",
        "        \"num_workers\": 1,\n",
        "        \"preprocess_config\": {\n",
        "            \"input_schema\": None,\n",
        "            \"output_schema\": None,\n",
        "            \"sequence_length\": 128,\n",
        "            \"first_sequence\": None,\n",
        "            \"second_sequence\": None,\n",
        "            \"label_enumerate_values\": None,\n",
        "        },\n",
        "        \"model_config\": {\n",
        "            \"pretrain_model_name_or_path\": None,\n",
        "            \"num_labels\": None\n",
        "        },\n",
        "        \"train_config\": {\n",
        "            \"keep_checkpoint_max\": 11,\n",
        "            \"save_steps\": None,\n",
        "            \"optimizer_config\": {\n",
        "                \"optimizer\": \"adam\",\n",
        "                \"weight_decay_ratio\": 0.01,\n",
        "                \"warmup_ratio\": 0.1,\n",
        "            },\n",
        "            \"distribution_config\": {\n",
        "                \"distribution_strategy\": None,\n",
        "            }\n",
        "        },\n",
        "        \"evaluate_config\": {\n",
        "            \"eval_batch_size\": 8\n",
        "        },\n",
        "        \"predict_config\": {\n",
        "            \"predict_checkpoint_path\": None,\n",
        "            \"predict_input_fp\": None,\n",
        "            \"predict_output_fp\": None,\n",
        "            \"predict_batch_size\": 1\n",
        "        },\n",
        "        \"worker_hosts\": \"localhost\",\n",
        "        \"task_index\": 1,\n",
        "        \"job_name\": \"chief\",\n",
        "        \"num_gpus\": 1,\n",
        "        \"num_workers\": 1,\n",
        "        \"model_config\": {\n",
        "            \"pretrain_model_name_or_path\": pretrain_model_name_or_path,\n",
        "            \"num_labels\": None\n",
        "        },\n",
        "        \"train_config\": {\n",
        "            \"train_input_fp\": task_dir+ \"/train.csv\",\n",
        "            \"train_batch_size\": 16,\n",
        "            \"model_dir\": task_name + \"_model_dir\",\n",
        "            \"num_epochs\": 2,\n",
        "            \n",
        "            \"keep_checkpoint_max\": 11,\n",
        "            \"save_steps\": None,\n",
        "            \"optimizer_config\": {\n",
        "                \"optimizer\": \"adam\",\n",
        "                \"weight_decay_ratio\": 0.01,\n",
        "                \"warmup_ratio\": 0.1,\n",
        "                \"learning_rate\": 1e-5,\n",
        "            },\n",
        "            \"distribution_config\": {\n",
        "                \"distribution_strategy\": None,\n",
        "            }\n",
        "        },\n",
        "        \"evaluate_config\": {\n",
        "            \"eval_input_fp\": task_dir + \"/dev.csv\",\n",
        "            \"eval_batch_size\": 8\n",
        "        },\n",
        "    \n",
        "        \"predict_config\": {\n",
        "            \"predict_checkpoint_path\": best_ckpt_path,\n",
        "            \"predict_input_fp\": task_dir + \"/test.csv\",\n",
        "            \"predict_output_fp\": None,\n",
        "            \"predict_batch_size\": 1\n",
        "        }\n",
        "    }"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "389f4616-59b4-4b5f-97bb-93c3eb2580f2",
        "id": "opVOXmAYSPVS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def task_config_json_predict(val,config_json):    \n",
        "    if val == \"TNEWS\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"115,114,108,109,116,110,113,112,102,103,100,101,106,107,104\"\n",
        "        config_json['model_config']['num_labels'] = 15\n",
        "\n",
        "    elif val == \"AFQMC\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"0,1\"\n",
        "        config_json['model_config']['num_labels'] = 2\n",
        "\n",
        "    elif val == \"IFLYTEK\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \",\".join([str(idx) for idx in range(119)])\n",
        "        config_json['model_config']['num_labels'] = 119\n",
        "\n",
        "    elif val == \"CMNLI\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"entailment,neutral,contradiction\"\n",
        "        config_json['model_config']['num_labels'] = 3\n",
        "\n",
        "    elif val == \"CSL\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"label:str:1,sent1:str:1,sent2:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['preprocess_config']['second_sequence'] = \"sent2\"\n",
        "        config_json['preprocess_config'][\n",
        "            'label_enumerate_values'] = \"0,1\"\n",
        "        config_json['model_config']['num_labels'] = 2\n",
        "    elif val == \"WSC\" or val == \"CLUEWSC\":\n",
        "        config_json['preprocess_config']['input_schema'] = \"idx:str:1,sent1:str:1\"\n",
        "        config_json['preprocess_config']['first_sequence'] = \"sent1\"\n",
        "        config_json['model_config']['num_labels'] = 2\n"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "4c9f5776-7a0a-4169-979b-0bced887b9e1",
        "id": "by5JXZ5bSPVT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "task_config_json_predict(task_name, predict_config_json)\n"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "55ffcba4-99cf-4977-b119-dd31a404c6fa",
        "id": "iyL-1VAwSPVV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7c61a5f8-0bc7-4a07-dcd2-4f428a40bfa3"
      },
      "source": [
        "config = Config(mode=\"predict_on_the_fly\", config_json=predict_config_json)\n"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:***************** modelZooBasePath /root/.eztransfer_modelzoo ***************\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0OvH014rnjAx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258
        },
        "outputId": "c4627cce-00cb-4e75-bb59-ea8c453198f4"
      },
      "source": [
        "app = Application(user_defined_config=config)\n",
        "\n"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:total number of predicting examples 290\n",
            "INFO:tensorflow:***********Running in predict_on_the_fly mode***********\n",
            "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpnsu9p7ue\n",
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpnsu9p7ue', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 1024\n",
            "inter_op_parallelism_threads: 1024\n",
            "gpu_options {\n",
            "  per_process_gpu_memory_fraction: 1.0\n",
            "  allow_growth: true\n",
            "  force_gpu_compatible: true\n",
            "}\n",
            "allow_soft_placement: true\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe70eed5ba8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
            "WARNING:tensorflow:Estimator's model_fn (<function EzTransEstimator._build_model_fn.<locals>.model_fn at 0x7fe71f89ed08>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LguFAdcHpIRl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "8d33ec0e-61c3-4cde-9bfe-62ba9c479fbd"
      },
      "source": [
        "pred_reader = CSVReader(input_glob=app.predict_input_fp,\n",
        "                        is_training=False,\n",
        "                        input_schema=app.input_schema,\n",
        "                        batch_size=1)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:num_parallel_batches 1\n",
            "INFO:tensorflow:shuffle_buffer_size None\n",
            "INFO:tensorflow:prefetch_buffer_size 1\n",
            "INFO:tensorflow:batch_size 1\n",
            "INFO:tensorflow:distribution_strategy None\n",
            "INFO:tensorflow:num_micro_batches 1\n",
            "INFO:tensorflow:input_schema idx:str:1,sent1:str:1\n",
            "INFO:tensorflow:./data/CLUEWSC/test.csv, total number of eval examples 290\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "uuid": "5984d974-45ab-479f-86a9-3c9c17a04836",
        "id": "Tf1JEI1FSPVX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "82919c0b-dda2-444b-ea1f-b888650306ae"
      },
      "source": [
        "id = 0\n",
        "if task_name == \"CLUEWSC\":\n",
        "  predict_prefix = \"wsc\"\n",
        "else:\n",
        "  predict_prefix = task_name.lower()\n",
        "\n",
        "with open(predict_prefix + \"_predict.json\", \"w\") as f:\n",
        "  for x in app.run_predict(reader=pred_reader,\n",
        "                         checkpoint_path=app.predict_checkpoint_path,\n",
        "                         yield_single_examples=True):\n",
        "    if id < 5:\n",
        "      print(\"id:\", id)\n",
        "    label = None\n",
        "    if x['predictions'] == 0:\n",
        "        label = \"true\"\n",
        "    else:\n",
        "        label = \"false\"\n",
        "    idx = str(x['predictions'])\n",
        "    f.write(\"{\\\"id\\\": \" + str(id) + \", \\\"label\\\": \" + \"\\\"\" + label + \"\\\"}\" + \"\\n\")\n",
        "    id += 1"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Load weights from /root/.eztransfer_modelzoo/bert/google-bert-base-zh/model.ckpt\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from ./data/CLUEWSC_model_dir/model.ckpt-77\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "id: 0\n",
            "id: 1\n",
            "id: 2\n",
            "id: 3\n",
            "id: 4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MI75ws9jVQCX",
        "colab_type": "text"
      },
      "source": [
        "## 最后从本地文件夹中找到对应的预测文件下载到本地，然后压缩成zip形式，提交到榜单上https://cluebenchmarks.com/"
      ]
    }
  ]
}
